White balance compensation using a spectral sensor system

ABSTRACT

A system for imaging a scene, includes a plurality of optical sensors arranged on an integrated circuit and a plurality of sets of interference filters, where each set of interference filters of the plurality of sets of interference filters includes a plurality of interference filters that are arranged in a pattern and each interference filter of the plurality of filters is configured to pass light in a different wavelength range, where each set of interference filters of the plurality of interference filters is associated with a spatial area of the scene. The system includes a plurality of rejection filters arranged in a pattern under each set of interference filters, where each rejection filter of the plurality of rejection filters is configured to substantially reject light of predetermined wavelengths. The system further includes one or more processors adapted to provide a spectral response for a spatial area of the scene associated with the set of interference filters.

CROSS REFERENCE TO RELATED PATENTS

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. § 119(e) to U.S. Provisional Application No. 63/066,507,entitled “WHITE BALANCE COMPENSATION USING A SPECTRAL SENSOR SYSTEM,”filed Aug. 17, 2020; and U.S.

Provisional Application No. 63/047,084, entitled “WHITE BALANCECOMPENSATION USING A SPECTRAL SENSOR SYSTEM,” filed Jul. 1, 2020, eachof which is hereby incorporated herein by reference in its entirety andmade part of the present U.S. Utility patent application for any and allpurposes.

BACKGROUND OF THE INVENTION Technical Field of the Invention

This invention relates generally to digital imaging and moreparticularly to compensating for light source distortion using spectralsensors with interference-based filters.

Digital imaging has had a profound effect on the quality andavailability of camera technology. At the same time, the expectations ofcamera consumers have become ever more demanding, especially for camerasembedded in modern smart phones. Automated white balancing, for example,has improved the quality of camera imaging by compensating for thedistorting effects of various light sources on a cameras output.

Spectroscopy devices, which function by detecting and/or acquiringincident light relating to multiple ranges of wavelengths, can be usedto provide spectral information to assist automated white balancing.Interference-based filters, such as Fabry-Perot filters, when used inconjunction with spectral sensors have been shown to be capable ofproviding information that can be used in a camera system to improveautomated white balancing.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 provides a top-down illustration of an example optical sensoroverlaid with filters in accordance with the present invention;

FIG. 2 provides a side illustration of adjacent Fabry-Perot filters withdifferent cavity thicknesses for an image sensor in accordance with thepresent invention;

FIG. 3 provides a side illustration of a pair Bragg stack mirrors inaccordance with the present invention;

FIG. 4 provides an illustration of interference filters and nearinfrared (NIR) filters combined to filter wavelengths in the infraredand visible spectrum in accordance with the present invention;

FIG. 5 provides a top-down illustration of a filter mosaic pattern for aspectral sensor in accordance with the present invention;

FIG. 6 provides another top-down illustration of a filter mosaic patternfor a spectral sensor in accordance with the present invention;

FIG. 7 provides another top-down illustration of a filter mosaic patternfor a spectral sensor in accordance with the present invention;

FIG. 8 provides a top-down illustration of an image sensor with astandard RGB mosaic pattern with one of the sensors replaced by aspectral filter element in accordance with the present invention;

FIG. 9 provides a cross-section of adjacent Fabry-Perot filters overlaidby a fiberoptic plate in accordance with the present invention;

FIG. 10 provides across-section of adjacent Fabry-Perot filters abovelight pipes in accordance with the present invention;

FIG. 11 provides a cross-section of adjacent Fabry-Perot filters with alight shield to isolate adjacent filters from crosstalk in accordancewith the present invention;

FIG. 12 provides a cross-section of adjacent Fabry-Perot filters with atrench used to isolate adjacent filters from crosstalk in accordancewith the present invention;

FIG. 13 provides a top-down illustration of a filter array with a shieldgrid in accordance with the present invention;

FIG. 14 provides a cross-section of adjacent Fabry-Perot filters with anisolation space between adjacent optical sensors in accordance with thepresent invention;

FIG. 15 provides an illustration of a filter structure that mirrors likefilter bands in adjacent filter mosaics in accordance with the presentinvention;

FIG. 16 provides an illustration of color matching functions for the CIEXYZ standard observer in accordance with the present invention;

FIG. 17 provides a top-down illustration of a CIE/XYZ mosaic structurein a Bayer pattern in accordance with the present invention;

FIG. 18A provides a cross-section of adjacent Fabry-Perot filtersoverlaid by an optical angular element in accordance with the presentinvention;

FIG. 18B illustrates a single optic is positioned over a sub-array of afilter array in accordance with the present invention;

FIG. 18C illustrates 3 optics is positioned over a different sub-arrayof a larger filter array in accordance with the present invention;

FIG. 18D provides a cross-section of interference filter sub-arrays withassociated optics in accordance with the present invention;

FIG. 19A illustrates an imaging system incorporating a high-resolutionand a low-resolution imager in accordance with the present invention;

FIG. 19B illustrates an imaging system incorporating a high resolutionwith two low-resolution imagers in accordance with the presentinvention;

FIG. 20 provides a top-down illustration of a pixel array with adjacentfilter mosaics on a sensor in accordance with the present invention;

FIG. 21 provides a block diagram of an imaging system incorporating ahigh-resolution image and a low-resolution imager in accordance with thepresent invention;

FIG. 22 is a flowchart illustrating an example method for correctinglight distortion in accordance with the present invention;

FIG. 23 is a flowchart illustrating another example method forcorrecting light distortion in accordance with the present invention;

FIG. 24 provides a top-down view of an optical sensor system usingoptical sensors/detectors comprising nanoscale semiconductor materialsin accordance with the present invention;

FIG. 25 provides a cross-section of adjacent Fabry-Perot filtersoverlaid by an optical angular element in accordance with the presentinvention;

FIG. 26 illustrates a scene with one or more light sources;

FIG. 27A is a flowchart illustrating an example method for collectinglight source information from a digital image of a scene in accordancewith the present invention;

FIG. 27B is another flowchart illustrating an example method forcollecting light source information from a digital image of a scene inaccordance with the present invention;

FIG. 28 is a flowchart illustrating an example method for compensatingfor ambient light flicker in a scene being captured by a digital imagingsystem in accordance with the present invention;

FIG. 29A illustrates the separate spectral responses for two spectralsensors (pixels) having adjacent central wavelengths;

FIG. 29B illustrates the combined spectral responses for the twospectral sensors;

FIG. 29C illustrates a pair of adjacent interference filters eachassociated with an optical sensor in accordance with the presentinvention;

FIG. 29D illustrates a pair of adjacent interference filters associatedwith a single optical sensor in accordance with the present invention;and

FIG. 29E illustrates a pair of interference filters placed one atop theother and associated with a single optical sensor.

DETAILED DESCRIPTION OF THE INVENTION

In various embodiments, spectral image sensors are combined withspectral filters such as interference-based interference filters toprovide spectral information about a scene and/or light source. In someembodiments, spectral imaging of a scene can be performed and in otherembodiments spectral imaging of a scene can either be combined with highresolution imaging in a single imager, or separate imagers combinedafter an image is collected. In further embodiments, interference-basedfilters can be implemented using Fabry-Perot filters integrated withspectral image sensors, such as CMOS-based sensors, to providesmall-scale spectral image sensor systems. In some embodiments,small-scale spectral imaging systems can be adapted for use inapplications that require white balance correction. Examples ofapplications include, but are not limited to, smart mobile phones, highresolution cameras, video cameras, security cameras, calibrationsystems, inspection systems and certain industrial applications.

Compensating for light source distortion, sometimes called “white-pointbalancing” is a fundamental part of a camera's rendering of images.Without white-point balancing an image sensor will not accuratelyrepresent the expected colorimetry of a recorded scene or object.Various light sources distort the colorimetry of objects in an imagesensor's field of view. For example, incandescent lights, fluorescentlights and light emitting diodes (LEDs) each distort the light that animage sensor “sees”. Other light sources, such as sodium street lights,distort an image sensor's output sufficiently that most colors arealmost virtually impossible to distinguish.

White balance compensation has provided the impetus for steady progress,eventually resulting in automatic white-balancing, which allowsphotographers to compensate for color imperfections resulting from lightsources at the output of an image sensor itself. In one example, an RGBoptical sensor, which is a semiconductor device that contains threetypes of pixels with peak sensitivity in the red, green and blue partsof the visible light spectrum, has been used to provide a reference forautomatic white-balancing. The combination of the red green and bluewavelengths of an RGB sensor appear to an observer to be “white”, thusin a scene containing one or more substantially white objects, the RGBsensor can combine the red green and blue wavelengths to appear to anobserver as white. Accordingly, in a scene containing such asubstantially white object, the RGB sensor can use the white object as areference point for adjusting the treatment of any other colors in ascene. AWB has evolved from combining the output of an RGB sensor on thecamera to use as a reference for white balancing to includemulti-channel spectral sensors. The accuracy of these multi-channelspectral sensors improve as more channels are distributed across thevisible light spectrum, however, in each case an imager with amulti-channel spectral sensor is limited to a single average referencespectrum to use for AWB of a given scene. Accordingly, in circumstanceswhere multiple light sources are present, or where a scene is dominatedby a single object, an image sensor can only compensate for the“average” illumination of a particular scene.

FIG. 1 provides a top-down illustration of a spectral sensor withfilters provisioned in a 3×3 patterns of 9 spectral bands each across animager array. In the example, Fabry-Perot filters with different centerwavelengths are patterned across the spectral sensor as a mosaicstructure repeated across the array. In other embodiments, the 3×3filter pattern can be replaced with other patterns, such as a 2×2pattern, a 4×4 filter pattern, a 5×5 filter pattern or a 3×4 pattern,etc., as dictated by resolution and/or manufacturing requirements. In anexample, a 3×3 pattern of filters provides 9 different cavitythicknesses, which are then repeated across an example sensor array. Inthe example of FIG. 1 each of the 9 filter thicknesses (illustrated asfilters 20A-20H, etc.) is repeated 12 times across the 12×9 array ofoptical pixels on sensor 10.

In the sensor system based on FIG. 1 optical pixels for sensor 10 aredisposed on an integrated circuit with a plurality of sets ofinterference filters manufactured on top of the optical pixels. In anexample, a set of nine (9) interference filters 20A-20I are arranged ina mosaic pattern, each of which is configured to pass light in adifferent wavelength range. In an example, each set of interferencefilters is aligned to at least a set of optical sensors, such that eachset of optical sensors is able to sense a localized bandpass responsewith 9 channels. The set of optical sensors and filter arrangement arethen repeated across the array, enabling the optical sensor array toprovide multiple measured light spectra spatially separated acrossdifferent areas of an image sensor. As used herein, an individualoptical sensor corresponds to a pixel (pixel=smallest addressableelement), where a pixel is a photodiode. Accordingly, “optical sensor”,“optical pixel” and “pixel” are used interchangeably.

In an example, the image sensor of FIG. 1 is able to provide lightdistortion information for different areas of the image sensor, allowingwhite-balance correction to be extended to each of those areas. In anexample of implementation, a sensor system for imaging a scene cancomprise a plurality of optical sensors on an integrated circuit, with aplurality of sets of interference filters, such as filter elements20A-20I of FIG. 1. In the example, each set of interference filters ofthe plurality of sets of interference filters can include a plurality ofinterference filters arranged in a pattern, where each interferencefilter of the plurality of filters is configured to pass light in adifferent wavelength range. In an example, each set of interferencefilters of the plurality of interference filters is associated with aspatial area of the scene and a spectral response can thus be determinedfor each spatial area of the scene.

In an example of implementation referring to FIG. 1, a set ofinterference filters of a plurality of sets of interference filters canbe spatially separate from others of the plurality of sets ofinterference filters and in another example, each set of interferencefilters of the plurality of sets of interference filters can spacedrandomly between the plurality of optical sensors of sensor 10.

FIG. 2 provides a cross-section of adjacent Fabry-Perot filter stacks(filters) with different cavity thicknesses for an image sensor, suchas, for example, the image sensor of FIG. 1. As illustrated, the centerwavelength of each Fabry-Perot filter is determined in first order bythe cavity thickness between its upper and lower mirror. In the example,adjacent filters 20A-20F provide 6 channels of sensor output. Betweenfilters 20A-20F and sensor 10, rejection filters 30A—30C are provided toblock stray light outside the desired wavelengths of the associatedinterference filters. In some circumstances a Fabry-Perot filter maypass wavelengths, such as harmonic wavelengths or wavelengths outsidethe valid range of the (Bragg) mirrors, that will negatively impact thedesired wavelength response of the filter. Accordingly, a rejectionfilter can act as a bandpass filter, rejecting wavelengths outside ofthe bandpass range. In an example, a single rejection filter may providesufficient bandpass rejection for two or more Fabry-Perot filters. Inanother example, rejection filters can be disposed above the associatedFabry-Perot filters to reject light outside of the desired wavelengthrange before it can be passed by the Fabry-Perot filters. In yet anotherexample, additional interference filters, such as Fabry-Perot filters,can be disposed between one or more rejection filters and the sensor 10.In the example, filters 20A-20F overlay one or more rejections filters,with the additional interference filters underlaying the one or morerejection filters.

In an example, rejection filters can comprise organic material and canbe applied using a spin-on process. In another example, rejectionfilters can comprise plasmonic interference filters applied by, forexample, a lithographic process. In another example, rejection filtersmay be colloidal or quantum dot-based filters. Other examples ofrejection filters include a combination of organic materials and/orplasmonic filters. And in yet another example, a rejection filter maycomprise one or more interference filters, either alone or incombination with organic materials and/or plasmonic filters. In anexample, a plurality of rejection filters can be arranged in a patternunder a mosaic of filter elements, where each rejection filter of theplurality of rejection filters is configured to substantially rejectlight of predetermined wavelengths.

In a specific example of implementation, a set of interference filtersis arranged in a pattern that further includes a plurality of organicfilters and in another example, the pattern includes a plurality ofnon-interference filters, wherein the non-interference filters areselected from a group that consists of organic filters, plasmonicfilters or a suitable alternative.

In a related example, a rejection filter can comprise a Bragg stackmirror. In the example illustrated in FIG. 3, a Bragg stack mirror actsas a rejection filter for Filter 20A and 20B, while acting as the Braggstack mirror from the Fabry-Perot filters 20C and 20D in FIG. 3. In yetanother example, one or more of the rejection filters can comprisemultiple thin layers of dielectric material, deposited and patterned,for example using a thin film deposition process and/or lithographicprocess. Accordingly, the patterning process can consist of lithographictreatment to define the filter spatial positions, combined with etchingor lift-off techniques to remove deposited filter layers locally.Specific etch-stop layers may be deposited in the filter stack tocontrol etch processes, allowing removal of optical layers in the filterstack locally. In an example, an etch-stop layer that does not affectoptical performance may be used to protect filter 20A and 20B from beingetched away while filter material from other locations is being removed.An etch-stop can be used when defining the bandpass filters as well asthe rejection filters.

In a specific example of implementation, one or more rejection filtersof a plurality of rejection filters is another interference filter. Inthe example, the another interference filter is one of the plurality ofinterference filters. In another example, the other interference filteris at the same time configured to pass light in a particular wavelengthrange and reject light for another optical sensor and interferencefilter pair.

FIG. 4 provides an illustration of interference filters used forfiltering visible light and combined with near infrared (NIR) filters tofilter wavelengths in the infrared spectrum. In an example, one or moreNIR filters can be composed of organic materials, while the interferencefilters comprise Fabry Perot filters, allowing the measurement of lightwavelengths across the visible and infrared spectrum. In the example ofFIG. 4, filters 50A-50C can be any of Fabry-Perot filters, organicfilters or any other acceptable alternative.

In an example, non-CMOS based optical sensors can be used to extend thespectral range of a spectral sensor to infrared wavelengths. Forexample, colloidal or quantum dot-based optical sensor may be used tocollect infrared light, for example in the short-wave infrared range. Inthe example of a quantum dot-based optical sensor, the optical sensorsmay be optimized by tuning the quantum dot size, such that a predefinedwavelength is selected, so that the optical sensor provides an infraredfilter channel. In another example, a sensor system can include aplurality of sets of optical sensors, wherein each set of opticalsensors is arranged in a pattern that includes at least one opticalsensor that is respectively larger in size than at least one otheroptical sensor of the set of optical sensors.

FIG. 5 provides a top-down illustration of a filter mosaic pattern for aspectral sensor that includes a large filter element. In the example, a6-filter mosaic includes standard filter elements 20B, 20C, 20D and 20Ewith a single filter element 22 that occupies the space of 4 standardfilter elements. In an example, the larger filter element 22 can providefor a 6-channel filter response in situations where some filter responserequirements dictate increased light capture, such as when a wavelengthrange requires a filter with reduced transmission properties. In aspecific example, a set of interference filters can be arranged in apattern that further includes an interference filter that isrespectively larger in size than at least one other interference filterin the set of interference filters.

FIG. 6 provides a top-down illustration of another filter mosaic patternfor a spectral sensor that includes filter elements forming largeroblong shapes. In the example, large filter element 24A and large filterelement 24B are included in a filter mosaic with 16 standard filterelements, such as filter elements 20A-20D. In an example, the inclusionof larger filter elements can provide for a 19-channel filter responsein situations where some filter response requirements dictate increasedlight capture, such as referenced with reference to FIG. 5. In anexample, a spectral filter mosaic can include an interference filterthat is respectively larger in size than at least one other interferencefilter in the set of interference filters and/or is in an elongatedrectangular shape.

FIG. 7 provides a top-down illustration of a filter mosaic pattern for aspectral sensor with filter elements forming progressively smaller ringsaround a central filter element. In the example, smaller filter element26D, is surrounded by larger filter element 26C, which is surrounded bean even larger filter element 26A, all of which are surrounded by largefilter element 26B. In an example, the progressively larger filterelements can provide for a 4-channel filter response in situations wheresome filter response requirements dictate increased light capture, suchas referenced with reference to FIG. 5. In an example spectral filtermosaic, one or more interference filters are respectively larger in sizethan at least one other interference filter in the set of interferencefilters and/or is adapted to form a ring around the other interferencefilters in the set of interference filters.

FIG. 8 provides a top-down illustration of an image sensor with astandard RGB mosaic pattern with one of the sensors replaced by aspectral filter element. In the example, pixel sensors 20A, 20B and 20Cform a 2×2 mosaic pattern that includes filter 32A (1). In an example,the standard RGB mosaic pattern is repeated across sensor 10, with each2×2 RGB mosaic including a spectral filter element, such as filterelements 32B and 32C of a multi-band spectral sensor. For example, thesensor 10 of FIG. 8 is an 8×8 array of sensors with 4×4 RGB mosaics thatinclude 4×4 spectral sensors. Accordingly, in the example of FIG. 8, thestandard 16 RGB array can include 16 spectral sensor channels for thesensor 10. In an example, the RGB and spectral sensor combination can berepeated across the spatial area of sensor 10 to provide localizedspectral response for a large image sensor.

In an example of implementation, A sensor system can include a pluralityof sets of optical sensors on an integrated circuit, where each set ofoptical sensors includes a plurality of optical sensors arranged in apattern. In the example, one or more sets of interference filters, eachof which includes a plurality of interference filters, each interferencefilter is located on top of an optical sensor of the plurality of a setsof optical sensors and each interference filter of a set of interferencefilters is configured to pass light of a different wavelength range. Ina specific example, the pattern for the set of optical sensors includes4 sections to form a 2×2 matrix, where each of a red, green and bluechannel sensor and a spectral channel sensor is located in one of the 4sections.

In a specific example of implementation, the pattern for the red, greenand blue channel sensors is a 2×2 pattern, while the pattern for thespectral sensors uses a repetition rate of N, where N>2 and the numberof different spectral sensors N>1. In another example, each colorchannel filter element and/or spectral channel filter for a sensorsystem covers more than one optical sensor in the pattern. In yetanother example, a filter pattern includes a set of color filtersintended for color imaging (such as red, green, blue, luminance, clear,etc.), such as that found in any modern imager and at least one set ofspectral filter elements.

In an example, different spectral filters of several of the patternstogether form a low-resolution spectral image of a scene, while thecolor filters of the pattern form a high-resolution color image of thescene. In a related example, the low-resolution spectral response isused to determining the white balance requirements of different spatialareas of the scene.

In a specific example of implementation, each interference filter of aset of interference filters is associated randomly with a spectralchannel sensor and in another example, the number of interferencefilters in each set of interference filters is different based on thespatial location of the set of interference filters in the sensorsystem. In yet another related example, the location of each set ofinterference filters and/or each interference filter in a spectralimager is based on a pseudo random pattern.

FIG. 9 provides a cross-section of adjacent Fabry-Perot filters 20A and20B overlaid by a fiberoptic plate 60. Referring back to FIG. 2, lightpassing through a filter, such as filter 20A of FIG. 2, at particularangles can be filtered by a particular filter while being detected by anoptical sensor associated with an adjacent filter. In a specificexample, filter 20A is configured to pass light of specific wavelengths,however, when the angle of incidence of the light passing through filter20A is sufficiently oblique, the light can propagate through theintegrated circuit back end 40 and be detected with an optical sensorassociated with filter 20B. Light of an undesired wavelength propagatingthrough an adjacent interference filter is often referred to as“crosstalk”. Crosstalk has an undesired effect on the quality of thespectral response of a filter mosaic, which in turn negatively impactsthe quality of light distortion corrections. Thus, eliminating or atleast attenuating the effects of crosstalk is desirable.

Fiberoptic plate 60 of FIG. 9 is an optical device comprised of a bundleof micron-sized optical fibers. When used as a lens on filters 20A and20B, light or an image transmitted through fiber optic plate iscollimated to reduce the angle-of-incidence (the angle between a rayincident on a surface and the line perpendicular to the surface at thepoint of incidence) of passing through the filters sufficiently toreduce unwanted crosstalk. Unlike a normal optical lens, no focusingdistance is required when using a fiber optic plate, such as fiber opticplate 60, accordingly it is compatible with compact optical devices.

FIG. 10 provides another cross-section of adjacent Fabry-Perot filters20A and 20B above light pipes 64. In the example of FIG. 10, light withtoo high angle-of-incidence passing through a filter is redirected bythe side walls of light pipes 64 to the optical sensors associated withthat filter. In a specific example, when the angle-of-incidence of lightpassing through filter 20A is sufficiently high, it will be reflectedoff the sidewall of light pipe 64 and be detected by an optical sensorassociated with filter 20A. In an example, the angle of side walls oflight pipes can be adjusted to provide a maximum attenuation whileminimizing absorption of desired light wavelengths. In an example, lightpipes 64 can be constructed of a variety of materials, where the lightpipe itself is a material with relatively high light transmission, withthe interstitial material being an opaque or semi-opaque material. Inanother example, the sidewalls of light pipes 64 can include arelatively high reflectivity material coated or deposited on it.

FIG. 11 provides another cross-section of adjacent Fabry-Perot filters20A and 20B with a light shield 68 to isolate adjacent filters 20A and20B from crosstalk. In the example of FIG. 11, light passing throughfilter 20A with excessive angle-of-incidence passing through a filter isdeflected or blocked by light shield 68. In a specific example, when theangle-of-incidence of light passing through filter 20A is sufficientlyhigh, it will either reflect off the side of light shield 68 or beblocked entirely, so that crosstalk to filter 20B will be eliminatedand/or attenuated. In an example, light shield 68 can be constructed ofa variety of materials, including opaque or semi-opaque material. Inanother example, the light shield 68 can be composed of metal, such asAl, or AlSi deposited in a trench formed and/or etched in the integratedcircuit back end 40 prior to addition of filters and/or rejectionfilters. In a specific example of implementation, metal is deposited onthe surface of integrated circuit back end 40 where trenches have beenformed and then removed from the areas outside the trenches using asubtractive process, such as chemical mechanical polishing and/or dryetching using a lithographic process. In another example, the depth andwidth of light shield 68 can be adjusted to provide attenuation ofparticular angles-of-incidence for more or less crosstalk attenuation aswarranted.

FIG. 12 provides another cross-section of adjacent Fabry-Perot filters20A and 20B with a trench 66 used to isolate adjacent filters 20A and20B from crosstalk. In the example of FIG. 12, light passing throughfilter 20A with excessive angle-of-incidence passing through a filter isdeflected or blocked by trench 66. In a specific example, when theangle-of-incidence of light passing through filter 20A is sufficientlyhigh, it will either reflect off the side of trench 66 or be blockedentirely, so that crosstalk to filter 20B will be eliminated and/orattenuated. In an example, trench 66 is formed and/or etched in theintegrated circuit back end 40 prior to addition of filters and/orrejection filters using a lithographic process. In an example, trench 66can be filled with another material or left as a void, with light beingeither reflected or refracted at the side walls of trench 66. In anotherexample, the depth and width of trench 66 can be adjusted to provideattenuation of particular angles-of-incidence for more or less crosstalkattenuation as warranted.

FIG. 13 provides a top-down illustration of filter array with a shieldgrid 110 to attenuate crosstalk between filter and optical sensor pairs.In the example of FIG. 13, incident light on filters 20A. 20D, 20E, etc.is blocked at shield grid 110 to provide a buffer zone between thefilters, such that the filters are at least partially isolated from eachother. In an example, shield grid 110 can be opaque material orsemi-opaque material or any other sufficiently absorptive materialdeposited or defined lithographically in the margins of filters 20A.20D, 20E, etc. In another example, shield grid 110 can be composed of areflective material, such as Al and/or AlSi. In an example, shield grid110 can be configured above or below filters 20A. 20D, 20E, etc.

In certain embodiments, an image sensor, such as sensor 10 of FIGS.9-13, can be configured to provide a dead space or void betweenindividual optical sensors and/or optical sensor components of an imagesensor. The dead space can provide isolation between the optical sensorsto reduce crosstalk between the optical sensors. In a related exampleillustrated in FIG. 14, an intermediate element 36 is located under theintersection of adjacent filters 20A and 20B and between photosensitiveelements 34. In an example, the intermediate element 36 is a dead spacebetween optical sensors of an image sensor. In another example, theintermediate element 36 and photosensitive elements 34 are all locatedin the dead space between optical sensors of an image sensor. In aspecific example of implementation, one or more responses fromphotosensitive elements 34 can be used to measure crosstalk and in arelated example, one or more responses from photosensitive elements 34can be used to correct the filter response for the measured crosstalk.

Referring to FIG. 1, a repeating mosaic pattern can necessarily maximizethe number of transitions between filter bands (where filters configuredto pass light in the same wavelength range are the same filter band).FIG. 15 provides an illustration of a filter structure that mirrors likefilter bands in adjacent filter mosaics in order to reduce the number oftransitions from one filter band to another. In the example, thepatterns for 4 three filter mosaics 1-4 are modified so that filters 20Aare adjacent to each other. In an example, crosstalk is reduced from atypical repeating pattern, because the number of transitions it reduced.

In a specific example of implementation, an example sensor system with 4sets of interference filters includes a plurality of sets ofinterference filters that include a plurality of interference filtersthat are arranged in a pattern, where the pattern for each of the 4 setsof interference filters is modified so that 4 interference filtersconfigured to pass light in the same wavelength range adjoin each otherat a quadripoint. In another specific example of implementation, 2 setsof interference filters of a plurality of sets of interference filtersinclude a plurality of interference filters that are arranged in apattern, where the pattern for each of the 2 sets of interferencefilters is modified so that 2 interference filters configured to passlight in the same wavelength range are adjacent to each other about acenterline between the 2 sets of interference filters.

In an embodiment, a sensor system includes a plurality of opticalsensors, one or more which are used for autofocusing. In a specificexample of implementation, a set of interference filters of a pluralityof sets of interference is adapted to locate a particular oneinterference filter of the plurality of interference filters atop theone or more optical sensors used for autofocusing.

In another embodiment, a sensor system includes a plurality of opticalsensors and a plurality of sets of interference filters that areprovisioned on the reverse side of the integrated circuit. In theexample, the reverse side of the integrated circuit is opposite a sideof the integrated circuit with wiring. In an example, the sensor systemcomprises a backside illumination image sensor. A back-illuminatedsensor, also known as backside illumination (BSI or BI) sensor uses thenovel arrangement of the imaging elements on the reverse side of theintegrated circuit comprising an image sensor in order to increase theamount of light captured and thereby improve low-light performance. Theincreased light capture is at least partially due to the fact that thematrix of individual picture elements and its wiring reflect some of thelight, and thus the sensor 10 can only receive the remainder of theincoming light, because the reflection reduces the signal that isavailable to be captured.

FIG. 16 provides an illustration of color matching functions for the CIEXYZ standard observer. (source:https://en.wikipedia.org/wiki/CIE_1931_color_space) The color matchingfunctions can be thought of as the spectral sensitivity curves of threelinear light detectors yielding the CIE tristimulus values X, Y and Z,where Y as luminance, Z is quasi-equal to blue, or the S cone response,and X is a mix of response curves chosen to be nonnegative. In anembodiment, the sensor system of FIGS. 1-16 include at least some of theplurality of interference filters of a set of interference filters areadapted to provide absolute-value color measurements (such as CIEtristimulus values X, Y and Z) when paired with an image sensor thatincludes plurality of optical sensors. In an example, absolute-valuecolor measurements are measurements include both a brightness and achromaticity.

FIG. 17 provides a top-down illustration of a CIE/XYZ mosaic structurein a Bayer pattern. In the example, interference filters 20A-20D arepatterned to form a true color sensor. The Bayer pattern (sometimesreferred to as a Bayer mosaic or Bayer filter mosaic) is an array forarranging color filters on a square grid of optical sensors.

FIG. 18A provides a cross-section of adjacent Fabry-Perot filters20A-20F overlaid by an optical element 80. In an example, an opticalelement is associated with one or more arrays of filters 20A-20F. FIGS.18B and 18C illustrate the incorporation of optics over sub arrays(bands) of a multi-spectral array. In FIG. 18B, a single optic 130 ispositioned over a sub-array or band of (filters 1-16) of a filter array120, while in FIG. 18C each of 3 optics 140, 142 and 144 is positionedover a different repeating sub-array. For example filter sub-array 124includes filters 1-9 (band 1), while filter sub-array 122 includesfilters 10-13 (band 3) and filter sub-array 126 includes filters 14-16(band 2) of a larger array. In a specific example of implementation, asensor system includes a plurality of optics over a plurality of opticalsensors, where each lens of the plurality of optics is associated withone or more sets of interference filters that are themselves associatedwith an optic of the plurality of optics. In an example, an opticcomprises a lens and a low-pass optical element. In an example, thelow-pass optical element of an optic is a diffuser and, in anotherexample, a low-pass optical element is located a predetermined distancefrom a plurality of sets of interference filters so that a blurred imageof predetermined blur dimensions is produced on the plurality of opticalsensors. In a different example, 2 or more of a plurality of optics,such as the 3 optics illustrated in FIG. 18C overlap a portion of alarger array, such that each of the 2 or more optics cover a portion ofthe larger array. In another specific example, the optical element 80can comprise an angular element, where the angular element is configuredto select an input angle for light propagating to one or more sensors.In yet another specific example, optical element 80 can comprise anoptical lens configured to rotate or tilt. Examples include opticalimage stabilization, lens rotation to change polarity of propagatinglight and/or another mechanical lens movement.

FIG. 18D is a cross-section of interference filter sub-arrays withassociated optics. In an example of implementation and operation, asystem includes a plurality of optical sensors on an integrated circuit,with a plurality of sets of interference filters, such as filters sets184A and 184B. In the example, a set of interference filters, such asfilters sets 184A and 184B are configured to pass light in a predefinedspectral range, with each interference filter of the plurality ofinterference filters configured to pass light in a different wavelengthrange. In an example, the system includes a one or more opticalelements, such as lenses 176A-176D where each optical element isassociated with at least one set of interference filters, such asfilters sets 184A and 184B to provide optics and interference filter setpairs. In another example of implementation, some of the interferencefilters of one or more sets of interference filters are Fabry-Perotfilters.

In an example of implementation, one or more optical elements includes afilter, such as filter 178 from FIG. 18D and a lens, such as lenses 176Cand 176D, to focus an image onto a group of pixels under filter set184B. In an example, filters 178A and 178B are rejection filters adaptedto reject unwanted out of band light. In another example, one or moreoptical elements includes more than one lens element, such as lenses176C and/or 176D. In an example, baffles 174 are configured to supportlenses 176A-176D, while isolating light incident on the pixels under agiven set of filters. In the example, each optical element andinterference filter pair comprises a sub-imager with pixels underlyingthe filter set, where multiple sub-imagers are likewise configured toprovide spectral information for a given scene in a different spectralrange. In yet another example, the optics is a plenoptic system.

In an example of implementation and operation, a first optical elementand interference filter pair is configured to pass light in theultraviolet (UV) spectrum, a second optical element and interferencefilter pair is configured to pass light in the infrared (IR) spectrum,and a third optical element and interference filter pair is configuredto pass light in the visible spectrum. In another example ofimplementation some of the optical sensors of a plurality of opticalsensors are not associated to any type of filter, allowing apanchromatic response.

In another example of implementation, rejection filters associated withoptical elements are integrated on the integrated circuit usingsemiconductor processing techniques. In another example, some or all ofthe elements of a plurality of optical elements are manufactured usingwafer-level optics, such as micro lenses.

In a specific example of implementation, a lens can be configured todefocus to produce a blurred image with predetermined blur dimensionsand then focus to produce a focused image at the plurality of opticalsensors. In a related example, the focused image is a high-resolutioncolor image, while the blurred image is a low-resolution color balancedimage. In another related example, a blurred image is used to provide arepresentative spectral response for the scene, where the representativespectral response includes a spectral response for a plurality ofspatial areas of the scene. In yet another example of implementation, anoptical lens is focused to form a high-resolution color image with thecolor sensors of an imager and defocused to form a low-resolution whitebalance image with the spectral sensors. Example optical lenses includecompound lenses, Fresnel lenses, multi-focal Fresnel lenses, molded lensarrays, etc., and can be mechanically and/or electronically focused. Thelenses can be integrated on silicon wafer during manufacture or they canbe coated and/or assembled on a finished image sensor. In an example,defocusing the optical lens can be done automatically when capturing animage, or manually with a user selecting a white-balance capture mode asneeded or desired.

FIG. 19A illustrates an imaging system incorporating a high-resolutionand a low-resolution imager, whereas FIG. 19B illustrates an imagingsystem incorporating a high resolution with two low-resolution imagers.In the example, spectral sensor(s) 170 is configured to provide a lowerresolution spectral image of a scene, while image sensor 172 isconfigured to provide a high-resolution image of the same scene. In anexample, the response from spectral sensor(s) 170 can be used to providecolor balancing of the spatial areas of a scene imaged with image sensor172. The imaging system can include one or more processors forprocessing the color balancing of the scene imaged with image sensor 172using the spectral responses from different spatial areas of the samescene.

In an example of implementation, a sensor system comprises a first groupof optical sensors associated with sets of interference filters, where aset of interference filters includes a plurality of interference filtersthat are arranged in a pattern. In an example, each interference filterof the plurality of filters is configured to pass light in a differentwavelength range and each set of interference filters of the pluralityof interference filters is associated with a spatial area of a scene. Inthe example, a second group of optical sensors is configured to outputan image; and one or more processors produce a spectral response for theplurality of spatial areas of the scene from the first group opticalsensors and an image is output by the second group of optical sensors.

In an example, a demosaicing process is used to extract the spectralbandpass response from a set of filters. The demosaicing process can beenabled using one or more processors, where the processors use analgorithm or digital image process to reconstruct a bandpass responsefrom optical sensors associated with individual filters of a set offilters. In an example where two groups of optical sensors areinterspersed a demosaicing process can be used to retrieve spectralinformation from a subset of filters in an interspersed group or array.

In an example of implementation, the second group of optical sensors isconfigured to produce a higher resolution image, while the first groupof optical sensors provides a lower resolution spectral response. Theone or more processors utilizes the lower resolution spectral responsefor at least some of the spatial areas of a scene to modify the higherresolution image of the scene based on the spectral response of theincluded spatial areas. In an example, the modification of the higherresolution image includes a color correction for the included spatialareas of the image of the scene.

In another example of implementation, one or more processors utilize thespectral response for the spatial areas of a scene to classify one ormore materials in the scene. In an example application, alow-spatial-resolution but high-spectral-resolution sensor image iscombined with a high-spatial-resolution but low-spectral-resolutionsensor image. In another embodiment, a first group of optical sensorsthat comprise the low-resolution spectral sensor provide spectralinformation of objects in a second group of optical sensors thatcomprise the high-resolution sensor. The spectral information caninclude information sufficient to determine properties of the object,such as material composition. The spectral information can furtherassist in identifying object types. Example applications can include,for example, skin sensing, water or oxygen detection, food detection,food analysis, quality inspection, plant analysis and dronesurveillance.

In an example of implementation, a first group of optical sensors and asecond group of optical sensors are adjacent to each other and inanother example, the first group of optical sensors being adapted foruse while in contact with one or more objects of a scene, while thesecond group of optical sensors is configured to not be in contact withthe one or more objects. In another example, the first group of opticalsensors and the second group of optical sensors are located on differentimage sensors. In yet another example, a first group of optical sensorsand a second group of optical sensors are on a common image sensor,where individual sensors of the first group of optical sensors aredistributed among optical sensors of the second group of opticalsensors. In a related example, a sensor system includes one or moreindividual optical sensors of a first group of optical sensors and aplurality of optical sensors of a second group of optical sensors, whereeach of the first group of optical sensors is associated with aplurality of optical sensors of the second group of optical sensors.

In a specific example of implementation, one or more processors are usedto approximate an output for one or more optical sensors of the secondgroup of optical sensors from an output of the first group of opticalsensors to produce an approximated output. And, in a further example,the approximated output is one of a red, green or blue sensor output. Inyet another example, an approximated output from an optical sensor ofthe first group of sensors is used to replace an output for an opticalsensor of the second group of optical sensors missing from a mosaicpattern of a subset of the second group of optical sensors. In anexample, the optical sensor may be missing due to, for example, anoptical sensor being replaced in the mosaic pattern with an opticalsensor used for spectral sensing.

In other examples, additional functionality is combined with the imagingsystem of FIGS. 19A and 19B. For example, the imaging system can beadapted to collect information produce a three-dimensional map of atleast a portion of the image, where, for example, the three-dimensionalmap can be used to determine approximate positions and interaction ofobjects in the scene. Further examples include adapting the imagingsystem to classify a material associated with one or more objects in thescene. The material can be classified based on illuminates, chromaticityor other phenomena.

In a related example, one or more 3D sensors adapted to output dataassociated with the scene can be included in an imaging system, wherethe imaging system is further adapted to produce a three-dimensional mapbased on the output data. In and additional example, the one or more 3Dsensors comprise a 3D point sensor and/or a 3D image sensor.

In other examples, of additional functionality combined with the imagingsystem of FIGS. 19A and 19B. the imaging system is adapted to collecttime-of-flight information for one or more objects in a scene where thetime-of-flight information is used to determine an approximate positionof one of more objects in the scene. In an example of implementation,the time-of-flight information is determined by modulating a signal anddetection for of one or more objects in the scene. In another example,time-of-flight information is used to further modify the image output bya high-resolution sensor, where the image output is further modified to,for example, correct for light distortion based on a distance from thesensor system. In the example, the imaging system may include variouslenses that can benefit from specific light distortion correction.Examples include, but are not limited to wide area lenses, ultrawidelenses, fish-eye lenses, telephoto lenses and others.

FIG. 20 provides a top-down illustration of a pixel array 172 withadjacent filter mosaics 182 on a sensor 10. In the pixel array 172 canbe a standard pixel array, with the filter mosaics 182 comprisingspectral sensors. In an example, N filter mosaics 182 can surround apixel array 172, where N is any of 2 to 4.

FIG. 21 provides a block diagram of an imaging system incorporating ahigh-resolution image 200, a low-resolution imager 210 and a 3-imageprocessor 220 adapted to produce a 4-image output 230. In an exampleembodiment, a camera system includes an image sensor, such ashigh-resolution image 200 that itself includes a number of sets ofoptical sensors, where each set of optical sensors includes a pluralityof optical sensors arranged in a pattern. The camera further includes aspectral image sensor, that itself comprises a plurality of opticalsensors, and a plurality of sets of interference filters, each of whichsets are arranged in a pattern. Each interference filter of theplurality of filters is configured to pass light in a differentwavelength range, and each set of interference filters of the pluralityof interference filters is associated with an area of a scene beingcaptured by the camera system. The camera system also includes one ormore processors that are adapted to produce a spectral response for aplurality of areas of the scene from the spectral image sensor andcombine the spectral response with an image output by the image sensorto produce a modified image.

In an embodiment of the camera system of FIG. 21, the modified image canbe an image that is at least partially corrected for light distortion inat least some of the plurality of areas of the image output. In anotherexample, the camera system is adapted to determine a light distortiontype for at least some of the plurality of areas of the image output.Examples of light distortion types include, but are not limited tonatural light, and various artificial light sources. The camera systemcan be further adapted to determine the frequency or duty cycle of alight source, such as whether a fluorescent light source is 50 Hz or 60Hz.

In another embodiment, the camera system of FIG. 21 can be adapted tomodify an image output, based on the spectral response, for display onone or more of a liquid crystal display, an organic light-emitting diodedisplay, a quantum dot display (QD-OLED) or a plasma display. In yetanother embodiment, the camera system can be adapted to modify the imageoutput, based on the spectral response, for use on a display lacking aninternal light source or on a display with a weak or selectable internallight source. Examples include displays that are used in high lightenvironments, such as outside or in locations with enough light (such asan event location or in office environments with sufficient artificiallight). In an example, a display that does not have a light source, butrather reflects light from outside the screen could be provided amodified image to optimize the display quality. Spectral response can,for example, be used to adjust a liquid crystal display to reflect acorrected color image.

In various embodiments, the camera system of FIG. 21, can be adapted todetermine a direction of illumination for one or more objects in a sceneand based on the direction of illumination, correct or otherwise enhancean image. Embodiments include, determining a light distortion type forat least some of the spatial areas of the image output and collectinginformation to produce a three-dimensional map of at least a portion ofthe image. In the example, any of direction of illumination, lightdistortion type(s) and the three-dimensional map can be used incombination to produce a further corrected and/or enhanced image. Yetother embodiments include determining a type of illumination for one ormore objects in a scene and based on the type of illumination,correcting an image for light distortion, where the type of illuminationcan include one or more of white LED illumination, color LEDillumination, phosphorus light source illumination, halogen light sourceillumination and various types of natural (sunlight) illumination.

In other embodiments, the camera system of FIG. 21, can include one ormore intelligent agents that are capable of cognitive functions, suchthat the intelligent agents are at least partially used to produce amodified/corrected image. In another example, where the camera system ofFIG. 21 includes one or more intelligent agents, wherein an intelligentagent is capable of cognitive functions, the intelligent agents can beused, for example, to determine a direction of illumination, produce athree-dimensional map and/or determine a light distortion type.

FIG. 22 is a flowchart illustrating an example method for correctinglight distortion in a scene by an imaging system. The method includes astep 300, where a sample light spectrum is received from local sets ofspectral sensors for the scene and continues at step 310, where anaverage spectral response is determined for each of the sample lightspectra. The method continues with the imaging system collecting imagedata from an image sensor for the scene at step 320 and at step 330 theimaging system corrects the image data for each respective local areasof the scene using the averaged spectral response for the local area.

FIG. 23 is a flowchart illustrating an example method for modifyingand/or correcting an image of a scene by an imaging system. The methodincludes a step 400, where a sample light spectrum is received fromlocal sets of spectral sensors for the scene and continues at step 410,where an illumination type is determined for each local area of thescene. The method continues at step 420, where the imaging systemproduces a 3D map of the scene and continues at step 430 where adirection of illumination is determined for one or more objects in thescene. The method continues with the imaging system collecting imagedata from an image sensor for the scene at step 440 and at step 450 theimaging system corrects the image data for each respective local areasof the scene based on the illumination type, direction of illuminationand 3D map to produce a corrected image.

FIG. 24 provides a top-down view of an optical sensor system usingoptical sensors/detectors comprising nanoscale semiconductor materials.Nanoscale semiconductor material based detectors, such as thin-filmquantum dot photodiodes can be manufactured using narrow bandgapthin-films compatible with conventional semiconductor processing. In anexample of implementation, the optical sensor system 10 incorporatesthin-film quantum dots 120 of varying size in order to provide spectralresponses across a predetermined spectrum, where the granularity andspectrum bandwidth of the thin-film is determined by the number and sizeof the quantum dots. The quantum dots can be, but are not limited toeither epitaxial quantum dots and/or colloidal quantum dots.

In a specific example of implementation and operation, a sensor systemcomprises a plurality of nanoscale semiconductor sensors configured tosense light in different wavelength bands on an integrated circuit. Inan example, the sensor system can be limited to optical sensorscomprising nanoscale semiconductors. In another example, the sensorsystem can include Fabry-Perot filters associated with CMOS opticalsensors. Nanoscale semiconductor elements can include one or more ofquantum dots, colloidal nanoparticles, CdSe nanocrystals and ZnSnanocrystals, etc. In a specific example of implementation, thenanoscale semiconductor elements can be implemented in different “dot”sizes, where the dot size dictates the wavelength of the spectralresponse for a given nanoscale element. In the example, various dotsizes are distributed on the sensor system in order to provide aspectrum of a given bandwidth and granularity.

FIG. 25 provides a cross-section of adjacent Fabry-Perot filtersoverlaid by an optical angular element in accordance with the presentinvention. In an example, an optical element 130 is associated with oneor more arrays of filters 20A-20F. In a specific example ofimplementation, optical element 130 can comprise an optical lensconfigured to rotate or tilt. Examples include optical imagestabilization, optical element rotation to change polarity ofpropagating light and/or another mechanical lens movement.

In another example of implementation and operation, the optical element80 of FIG. 2 comprises a plurality of integrated polarization elements(filters). In an example, the combination of polarization filters enablethe sensor system 10 to distinguish between or separate polarized lightand unpolarized light. In another example, the combination ofpolarization elements can enable the sensor system to separate lightinto different polarizations. In an example this polarizationinformation can be used to detect the illuminant spatial/directionalinformation and/or information about reflectance from objects in a sceneor image. In another example, the polarization information can be usedto detect glare and/or reflections in an image being imaged.

In a specific example of implementation and operation, a sensor system10 comprises a plurality of optical sensors on an integrated circuit,with a plurality of polarization filters disposed on top of the sensorsystem 10. In an example, the optical sensors can constitute an opticalsensor and polarization filter pair, and, in another example, multiplepolarization filters can be associated with a single optical sensor andin yet another example, a single polarization filter can be associatedwith multiple optical sensors. The sensor system 10 can include one ormore processors with one or more modules for determining polarizationinformation based on the output of the optical sensor with its pairedpolarization filter(s). In an example, the polarization information caninclude one or more of 1) illumination type for light detected by theoptical sensor of an optical sensor and polarization pair; 2) thedistribution of spatial information for light detected by the opticalsensor of an optical sensor and polarization pair; 3) directionalinformation for light detected by the optical sensor of an opticalsensor and polarization pair; and 4) reflection information for lightreflected by an object imaged by the sensor system.

In another specific example of implementation and operation, a sensorsystem 10 comprises a plurality of optical sensors on an integratedcircuit, with an optical system 80 disposed on top of the sensor system10. In the example the optical system can be used to select specificinput angles for light incident to the sensor system 10 and in anotherexample the input angles can be used to determine a location for lightsources in a scene being imaged. In an example, the optical system 80comprises a single optical element and in another example, it comprisesa plurality of optical elements.

In a specific example of implementation and operation, a sensor system10 comprises a plurality of optical sensors on an integrated circuit,with one or more optical elements 80 are located atop at least some ofthe plurality of optical sensors, where the one or more optical elements80 are configured to select input angles for light incident to thesensor system. In another example a processor can be used to determine adirection for light collected by one or more optical sensors of theplurality of optical sensors based on the selected input angles. In anexample, the determined light direction can be used to inform whitebalance modification or correction for a scene or object being imaged,where the white balance modification or correction is executed by aprocessor associated with the sensor system 10, or in an alternative,where the determined light direction is provided to an externalprocessing system for white balance modification or correction.

In another specific example of implementation and operation, opticalelement(s) 80 are common to all of the optical sensors in the sensorsystem 10 and in another example, the optical element(s) are common toonly a portion of the optical sensors in sensor system 10.

Several options are available for optical element(s) 130. In an example,optical element(s) 130 comprise optical lenses and in another examplethe optical element(s) 130 comprise one or more masks located proximateto the optical sensors, wherein a mask comprises a light shield with adifferent lateral offset for at least some of the optical sensors ofsensor system 10. In the example, each mask is configured to allow someincident angles of light while shielding other incident angles of light.The mask can be a single line of various materials, such as metal oranother opaque material or it can comprise a grid that is configured toprovide shielding for an array of optical sensors.

In another specific example of implementation and operation, opticalelement(s) 130 are optical micro-lenses; examples include, but are notlimited to, Fresnel lenses and/or molded lens arrays. In anotherspecific example, the optical element(s) 130 include mechanical elementsso that they can be rotated and/or tilted. In the example the opticalelement(s) 130 can be part of the optical image stabilization system fora camera incorporating the sensor system. In another specific example ofimplementation and operation, optical element(s) 130 are micro-lenses,where each micro-lens is adapted to select input angle(s) for one orsome portion of the optical sensors in the sensor system 10. In yetanother specific example of implementation and operation, opticalelement(s) 130 are polarization filters.

FIG. 26 illustrates a scene with one or more light sources. In thescene, light sources can illuminate a scene from behind, such as lightsource 140, and from the front and/or below, such as light source 142.In an example, white balance information can be included with digitalimaging data, allowing for the use of the white balance information inpost-processing of an image or video. In a specific example, in order toprovide for realistic lighting of an object added to the image, such asin an augmented reality system, illumination of the object can beadjusted to match the illumination of the scene. In the example similarto FIG. 25, an object (in this example a plastic shark) is added to apre-existing image and localized white balance information from thepre-existing image can be used to provide realistic lighting byilluminating the shark consistent with the pre-existing image. In anaugmented reality application, the illumination and/or shading of one ormore objects can be adjusted so as to make the objects morerealistically represented in the scene. Additionally, light directioninformation, such as that referred to in FIG. 25 can be used to informillumination of an object added to the pre-existing scene.

In a specific example of implementation and operation, a sensor systemis used to collect spectral information, such as white balanceinformation, from a scene. The sensor system can comprise a plurality ofoptical sensors with a plurality of sets of interference filters. In anexample, a set of interference filters of the plurality of sets ofinterference filters can be arranged in a pattern, wherein eachinterference filter of the plurality of filters is configured to passlight in a different wavelength range, with each set of interferencefilters associated with a spatial area of the scene. In example ofimplementation, the sensor system can include one or more processorsadapted to provide a spectral response based on an output from theplurality of optical sensors and determine spatial areas of the scenethat potentially represent sources of light based on the spectralresponse from each of the plurality of spatial areas of the scene.

In an example, the one or more processors can be adapted to identify thespatial areas of the scene that represent sources (and intensity) oflight for use to light one or more objects added after the digital imageof the scene is collected. In a related example, information associatedwith the spatial areas of the scene that represent sources of light canbe embedded in the digital image, provided as an addendum to the digitalimage and/or provided as a supplemental data file.

FIG. 27A is a flowchart illustrating an example method for collecting adigital image of a scene. The method starts at step 460, where receivedlight spectra from a scene are received from sets of spectral sensorsthat are spatially separated from each other. In an example, each set ofspectral sensors is associated with an image sensor. In an alternativeexample, each set of spectral sensors is associated with a spatial areaof the scene that is associated with a complementary spatial area of animage sensor. The method continues at step 464 with the lightcharacteristics of the light spectra are classified for each of thespatial areas of the image sensor. The method then continues at step462, with the received light spectra being used to determine whether oneor more sets of spectral sensors indicate a light source in the scene.When no light sources are detected, average white balance (AWB) for thescene is adjusted normally at step 468. In an example, the relativeintensity of the light source(s) can be determined and in anotherexample, when a light source is not located within the scene, thelocation and/or intensity of a light source outside of the scene can bedetermined based on reflections from objects within the scene. Inanother example, the light source location can be used to determineareas of the scene being imaged and/or objects in the scene that are ina shade. At an optional step 466, the spatial areas associated with thelight sources are identified. The method then continues at step 470 thelight source information, including the spatial area of the light sourceand a classification type are provided for the digital image. In anexample, the light source information can be used to adjust the whitebalance and or color homography for spatial areas of the scene.

FIG. 27B is another flowchart illustrating an example method forcollecting a digital image of a scene. The method starts at step 480,where received light spectra from a scene are received from sets ofspectral sensors that are spatially separated from each other. In anexample, each set of spectral sensors is associated with an imagesensor. In an alternative example, each set of spectral sensors isassociated with a spatial area of the scene that is associated with acomplementary spatial area of an image sensor. The method then continuesat step 482, with the received light spectra being used to determinewhether one or more sets of spectral sensors indicate a light source inthe scene. When no light sources are detected, average white balance(AWB) for the scene is adjusted normally at step 488. The methodcontinues at step 484 with the light characteristics of the lightspectra being classified for each of the spatial areas of the imagesensor. In an example, the relative intensity of the light source(s) canbe determined and in another example, when a light source is not locatedwithin the scene, the location and/or intensity of a light sourceoutside of the scene can be determined based on reflections from objectswithin the scene. In another example, the light source location can beused to determine areas of the scene being imaged and/or objects in thescene that are in a shade. At an optional step 486, the spatial areasassociated with the light sources are identified. The method thencontinues at step 490 the light source information, including thespatial area of the light source and a classification type are providedfor the digital image.

In a specific example of implementation and operation, the lightinformation provided in step 470 and step 490 of FIGS. 27A and 27B,respectively, can be used to aid in post processing of the digitalimage. In another example, the light information can be used to adjustthe illumination of objects added to the digital image in postprocessing. For example, once the spatial areas and associated lightsources are classified, that information can be provided in addition tothe captured image, so that it is available to be used in postprocessing when an object is subsequently added to the captured image.Referencing the example illustrated in FIG. 26, the plastic shark can beadded to the captured image of the room, with the light informationbeing used in post processing to properly place the light intensity andcoloration on the plastic shark. In yet another example, the lightinformation can be used to adjust the illumination of the scene capturedin the digital image to provide desired effects on the scene, such asimproving aesthetics of the scene or to artificially change theaesthetics of the scene. In an example of implementation, theillumination of a scene can be adjusted to provide a more natural effectusing the light information provided in step 470 and step 490 of FIGS.27A and 27B, respectively. In an alternative implementation, theillumination of a scene can be adjusted to augment or refine specialeffects using the light information provided in step 470 and step 490 ofFIGS. 27A and 27B, respectively

In a specific example of implementation and operation, light informationcan be provided while video imaging of a scene or object, so that thecaptured video can be substantially corrected in post processing. In anexample, each frame of the captured video can include at least somelight information. In another example, the light information can beprovided with the video imaging data on an intermittent basis (asopposed to frame-by-frame) so that the captured video can be correctedon a frame by frame basis by interpolating the light information missingfrom the frames without light information data. In yet another example,light source information, such as classification and/or intensity can belinked to objects or scene portions during the video coding of theimaged scene, so that the light information can be ignored until a sceneor object moves or changes, thereby enabling improved compression and/orreduced computation complexity of a captured video or image. In yetanother specific example, a video capture system can be adapted toinclude light information only when toggled on by a user, so that lightinformation, such light as classification and/or intensity would not beprocessed and/or captured when toggled off.

In an example, a camera system is adapted to determine a lightdistortion type for at least some of the plurality of areas of an imageoutput. (examples of light distortion types include, but are not limitedto natural light, and various artificial light sources.) The camerasystem can be further adapted to determine the frequency or duty cycleof a direct light source and/or ambient lights source, such as whether afluorescent light source is 50 Hz or 60 Hz. In yet another example, thecamera system can be further adapted to lock a negative feedbackcompensation loop to match the frequency or duty cycle and/or phase ofthe light source and then attenuate and/or cancel the resultant flickerof the light source. In an example, an optical amplifier can be used tocompensate for frequency effects by modifying the gain and phasecharacteristics of the amplifier's open loop output or of its feedbacknetwork, or both, in order compensate for the conditions leading tooscillation. In an example, the locked negative feedback compensationloop for a flicker disturbance can be provided to a plurality of (orall) affected pixels of a camera system, avoiding the saturation ofthose pixels by the flicker disturbance.

FIG. 28 is a flowchart illustrating an example method for compensatingfor ambient light flicker in a scene being captured by a digital imagingsystem. The method starts at step 500, where received light spectra froma scene are received from sets of localized spectral sensors. In step510 an illumination type is determined for the scene and at step 520 thedigital imaging system determines whether ambient light flicker isdetected in the scene. If there is no light flicker detected the digitalimaging system can adjust white balance for the scene at step 550 andwhen light flicker is detected the digital imaging system can use anegative feedback loop at step 530 to lock to the frequency and phase ofthe flicker source in order to compensate for the light flicker at step540.

FIG. 29A illustrates the separate spectral responses for two spectralsensors (pixels) having adjacent central wavelengths, such as spectralsensors based on Fabry-Perot filters, typically provide spectralresponses in narrow bands (such as 10 nm). Sometimes a spectral responsein a wider band (such as 20 nm or 30 nm) is desired. FIG. 29Billustrates the combined spectral responses for the two spectral sensorsof FIG. 29A, effectively doubling the spectral response bandwidth. In anexample of implementation and operation, a sensor system includes anarray of optical sensors and a plurality of spectral filters arranged inan array proximate to the array of optical sensors. In an example, thespectral filters are interference filters, such as Faby-Perot filters orplasmonic interference filters and organic filters. In an example, eachoptical sensor is associated with one or more spectral filters of theplurality of spectral filters, where each spectral filter of theplurality of spectral filters is configured to pass light of a selectedwavelength range. In an example, electronic circuitry is coupled to theoptical sensors, so that the output of two or more optical sensors canbe combined.

FIG. 29C illustrates a pair of adjacent interference filters (1) eachassociated with an optical sensor. In an example, the interferencefilters (1) are configured to pass wavelengths in adjacent ranges of alight spectrum. In an example, the output of the adjacent sensors (2) iscombined to produce a combined output with wider spectral responsebandwidth, as illustrated in FIG. 29B. In an example, the output of theadjacent sensors (2) can be combined using dedicated electroniccircuitry, or, in another example, it can be combined by one or moremodules of a computing device.

FIG. 29D illustrates a pair of adjacent interference filters (1)associated with a single optical sensor (2). In an example, incidentlight passing through both of the adjacent filters (1) is detected bythe single optical sensor (2). In an example of implementation andoperation, a sensor system includes an array of optical sensors and aplurality of spectral filters arranged in an array proximate to thearray of optical sensors. In an example, the spectral filters areinterference filters, such as Faby-Perot filters or plasmonicinterference filters and organic filters. In a specific example, anoptical sensor is associated with two or more spectral filters of theplurality of spectral filters, where each spectral filter of theplurality of spectral filters is configured to pass light of a selectedwavelength range. In the example, the two or more spectral filters ofthe plurality of spectral filters are configured to pass light insubstantially adjacent wavelength ranges, so that sensor 2 receiveseffectively double the wavelength of either of the interference filters(1) alone.

FIG. 29E illustrates a pair of interference filters (1) placed one atopthe other and associated with a single optical sensor (2). In an exampleof implementation and operation, a sensor system includes an array ofoptical sensors and a plurality of spectral filters arranged in an arrayproximate to the array of optical sensors. In an example, the spectralfilters are interference filters, such as Fabry-Perot filters orplasmonic interference filters. In a specific example, a first array ofspectral filters is arranged in an array over an array of opticalsensors, where each optical sensor of array of optical sensors isassociated with at least one spectral filter of the first array ofspectral filters, and each spectral filter is configured to pass lightof a selected wavelength range. In an example, a second array ofspectral filters is arranged in an array atop the first array ofspectral filters to produce spectral filter pairs, where each spectralfilter of the second array of spectral filters is configured to passlight of a selected wavelength range, where each spectral filter pair ofthe plurality of spectral filter pairs includes two or more spectralfilters that together are configured to pass light in substantiallyadjacent wavelength ranges. In an example, each filter pair isconfigured to pass light to a single optical sensor.

As discussed above, compensating for light source distortion usingautomatic white balance (AWB) correction enables an image sensor formore accurately representing the expected colorimetry of a recordedscene or object. In an example of implementation and operation, uniformAWB correction can be enhanced by blurring and/or scrambling a scenespatially when an imager is receiving input for AWB correction. Theblurred image can provide more uniform color detection for de-mosaicinga given set of spectral filter responses.

In an example of implementation and operation, a sensor system forimaging a scene, can include a plurality of optical sensors on anintegrated circuit with a plurality of sets of interference filters,where each set of interference filters includes interference filtersarranged in a pattern. Each interference filter of the plurality offilters is configured to pass light in a different wavelength range, andeach set of interference filters of the plurality of interferencefilters is associated with a spatial area of the scene. In an example, alens system is configured atop the plurality of optical sensors, wherethe lens system is adapted to produce a blurred image with substantiallypredetermined blur dimensions at the plurality of optical sensors. In anexample, the lens system is configured to defocus to produce the blurredimage with substantially predetermined blur dimensions and also focus toproduce a substantially focused image at the plurality of opticalsensors. In a specific example, the lens system is made of multipleelements, and the lens system is configured to defocus by adjusting onemore element of the multiple elements while not adjusting other elementsof the one or more elements.

In another specific example, the lens system can be adapted to introducespherical aberrations and/or other coherency aberrations to increase theblurring of a scene for AWB correction purposes. In yet another specificexample, the lens system can comprise a large field of view (FOV) andlow chief ray angles. The large field of view enables a given imager todetect additional light and capture a broad scene, while the low chiefray angles reduce the incident angles for incident light reaching thespectral filters, such as interference-based filters. FIG. 7 illustratesa lens system with a reverse telecentric design. The reverse telecentricdesign provides a large field of view and a low chief ray angle, suchthat by adjusting the elements of the telecentric design an image can beblurred for AWB correction while being adjustable to focus the image forhigh-spatial resolution image capture. Telecentric lenses are known toprovide an orthographic projection, providing the same magnification atall distances. An object that is too close may still be out of focus,but the resulting blurry image will have the same size as the correctlyfocused image would. In a reverse telecentric lens system one or moreelements of a telecentric lens system are reversed, resulting in a moreuniform color distribution to the spectral filters.

Electronics manufacturers increasingly utilize displays with underlyingimage sensors in smartphones, tablets, and other mobile devicesutilizing cameras. When image sensors are under a display, spectralrecoloring of an image can result, due at least in part to the activecolors emitted by the display corrupting the image. Spectral sensors canbe implemented under a display in order to mitigate the impact of thedisplay on an imager located under the display while also providinginput for automatic white balancing (AWB). In a specific example ofimplementation and operation, a sensor system for imaging a sceneincludes a first array of optical sensors and a plurality of sets ofinterference filters associated with the first array of optical sensors.Each set of interference filters of the plurality of sets ofinterference filters includes a plurality of interference filters thatare arranged in a pattern, where each interference filter of theplurality of interference filters is configured to pass light in adifferent wavelength range. Each set of interference filters of theplurality of interference filters is associated with a spatial area ofthe scene. In an example, the sensor system includes a second array ofoptical sensors that are configured to output an image and a processorwith one or more modules adapted to produce a spectral response for aplurality of spatial areas of the scene from the first array opticalsensors and an image output by the second array of optical sensors. Inthe example, a display is located atop the first plurality of opticalsensors and the second plurality of optical sensors.

Spectral sensors are presented in general as a means to improve thesignals from image sensors located under displays. Although in theexample given the spectral sensor and the image sensor are presented asseparate entities, a hyperspectral camera can implement both functions(spectral measurements and imaging) with the same optical sensor(s).

In a specific example of operation, the first array of optical sensorsand the second array of optical sensors are adjacent to each other underthe display. In the example, spectral response provided by the firstarray of optical sensors can be used to correct for light distortion andother artifacts for a scene being imaged by the second array of opticalsensors. In another example, the second array of optical sensors outputsa monochromatic image, while the output from the first array of opticalsensors can be used to provide color information for the monochromaticimage.

In another example, a portion of the optical sensors from the firstarray of optical sensors can be used to correct for the interferencefrom the display on an image generated by the second array of opticalsensors, with another portion of the optical sensors from the firstarray of optical sensors being available to provide color informationfor automatic white balancing (AWB). In a related example, opticalsensors from the first array associated with interference filtersconfigured to pass light in a certain wavelength ranges may be used tocorrect for display interference, while optical sensors from the firstarray associated with interference filters configured to pass light in aother wavelength ranges are used to correct for automatic whitebalancing (AWB). In an example, the processor can be further adapted todetect a change over time on the display colorimetry based on an outputfrom the display and/or the spectral response for the plurality ofspatial areas.

It is noted that terminologies as may be used herein such as bit stream,stream, signal sequence, etc. (or their equivalents) have been usedinterchangeably to describe digital information whose contentcorresponds to any of a number of desired types (e.g., data, video,speech, text, graphics, audio, etc. any of which may generally bereferred to as ‘data’).

As may be used herein, the terms “substantially” and “approximately”provide industry-accepted tolerance for its corresponding term and/orrelativity between items. For some industries, an industry-acceptedtolerance is less than one percent and, for other industries, theindustry-accepted tolerance is 10 percent or more. Other examples ofindustry-accepted tolerance range from less than one percent to fiftypercent. Industry-accepted tolerances correspond to, but are not limitedto, component values, integrated circuit process variations, temperaturevariations, rise and fall times, thermal noise, dimensions, signalingerrors, dropped packets, temperatures, pressures, material compositions,and/or performance metrics. Within an industry, tolerance variances ofaccepted tolerances may be more or less than a percentage level (e.g.,dimension tolerance of less than +/−1%). Some relativity between itemsmay range from a difference of less than a percentage level to a fewpercent. Other relativity between items may range from a difference of afew percent to magnitude of differences.

As may also be used herein, the term(s) “configured to”, “operablycoupled to”, “coupled to”, and/or “coupling” includes direct couplingbetween items and/or indirect coupling between items via an interveningitem (e.g., an item includes, but is not limited to, a component, anelement, a circuit, and/or a module) where, for an example of indirectcoupling, the intervening item does not modify the information of asignal but may adjust its current level, voltage level, and/or powerlevel. As may further be used herein, inferred coupling (i.e., where oneelement is coupled to another element by inference) includes direct andindirect coupling between two items in the same manner as “coupled to”.

As may even further be used herein, the term “configured to”, “operableto”, “coupled to”, or “operably coupled to” indicates that an itemincludes one or more of power connections, input(s), output(s), etc., toperform, when activated, one or more its corresponding functions and mayfurther include inferred coupling to one or more other items. As maystill further be used herein, the term “associated with”, includesdirect and/or indirect coupling of separate items and/or one item beingembedded within another item.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., provides a desiredrelationship. For example, when the desired relationship is that signal1 has a greater magnitude than signal 2, a favorable comparison may beachieved when the magnitude of signal 1 is greater than that of signal 2or when the magnitude of signal 2 is less than that of signal 1. As maybe used herein, the term “compares unfavorably”, indicates that acomparison between two or more items, signals, etc., fails to providethe desired relationship.

As may be used herein, one or more claims may include, in a specificform of this generic form, the phrase “at least one of a, b, and c” orof this generic form “at least one of a, b, or c”, with more or lesselements than “a”, “b”, and “c”. In either phrasing, the phrases are tobe interpreted identically. In particular, “at least one of a, b, and c”is equivalent to “at least one of a, b, or c” and shall mean a, b,and/or c. As an example, it means: “a” only, “b” only, “c” only, “a” and“b”, “a” and “c”, “b” and “c”, and/or “a”, “b”, and “c”.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, “processing circuitry”, and/or “processing unit”may be a single processing device or a plurality of processing devices.Such a processing device may be a microprocessor, micro-controller,digital signal processor, microcomputer, central processing unit, fieldprogrammable gate array, programmable logic device, state machine, logiccircuitry, analog circuitry, digital circuitry, and/or any device thatmanipulates signals (analog and/or digital) based on hard coding of thecircuitry and/or operational instructions. The processing module,module, processing circuit, processing circuitry, and/or processing unitmay be, or further include, memory and/or an integrated memory element,which may be a single memory device, a plurality of memory devices,and/or embedded circuitry of another processing module, module,processing circuit, processing circuitry, and/or processing unit. Such amemory device may be a read-only memory, random access memory, volatilememory, non-volatile memory, static memory, dynamic memory, flashmemory, cache memory, and/or any device that stores digital information.Note that if the processing module, module, processing circuit,processing circuitry, and/or processing unit includes more than oneprocessing device, the processing devices may be centrally located(e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that if the processing module, module, processing circuit,processing circuitry and/or processing unit implements one or more ofits functions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element may store, and the processing module, module,processing circuit, processing circuitry and/or processing unitexecutes, hard coded and/or operational instructions corresponding to atleast some of the steps and/or functions illustrated in one or more ofthe Figures. Such a memory device or memory element can be included inan article of manufacture.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with one or more other routines. In addition, a flow diagrammay include an “end” and/or “continue” indication. The “end” and/or“continue” indications reflect that the steps presented can end asdescribed and shown or optionally be incorporated in or otherwise usedin conjunction with one or more other routines. In this context, “start”indicates the beginning of the first step presented and may be precededby other activities not specifically shown. Further, the “continue”indication reflects that the steps presented may be performed multipletimes and/or may be succeeded by other activities not specificallyshown. Further, while a flow diagram indicates a particular ordering ofsteps, other orderings are likewise possible provided that theprinciples of causality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

Unless specifically stated to the contra, signals to, from, and/orbetween elements in a figure of any of the figures presented herein maybe analog or digital, continuous time or discrete time, and single-endedor differential. For instance, if a signal path is shown as asingle-ended path, it also represents a differential signal path.Similarly, if a signal path is shown as a differential path, it alsorepresents a single-ended signal path. While one or more particulararchitectures are described herein, other architectures can likewise beimplemented that use one or more data buses not expressly shown, directconnectivity between elements, and/or indirect coupling between otherelements as recognized by one of average skill in the art.

The term “module” is used in the description of one or more of theembodiments. A module implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A module may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a module may containone or more sub-modules, each of which may be one or more modules.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, or a set of memory locations within amemory device. Such a memory device may be a read-only memory, randomaccess memory, volatile memory, non-volatile memory, static memory,dynamic memory, flash memory, cache memory, and/or any device thatstores digital information. The memory device may be in a form asolid-state memory, a hard drive memory, cloud memory, thumb drive,server memory, computing device memory, and/or other physical medium forstoring digital information.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. A system for imaging a scene, comprising: a plurality of optical sensors on an integrated circuit; a plurality of sets of interference filters, wherein a set of interference filters of the plurality of sets of interference filters includes a plurality of interference filters that are arranged in a pattern, wherein each interference filter of the plurality of interference filters is configured to pass light in a different wavelength range, wherein each set of interference filters of the plurality of sets of interference filters is associated with a spatial area of the scene; and one or more processors, wherein the one or more processors are adapted to provide a spectral response for a spatial area of the scene.
 2. The system of claim 1, further comprising: a plurality of rejection filters arranged in a pattern, wherein the plurality of rejection filters is associated with one or more sets of interference filters, wherein each rejection filter of the plurality of rejection filters is configured to substantially reject light of one or more predetermined wavelengths.
 3. The system of claim 2, wherein the plurality of optical sensors, the plurality of sets of interference filters and the plurality of rejection filters are formed on a backside of an integrated circuit.
 4. The system of claim 2, wherein one or more rejection filters of the plurality of rejection filters is another interference filter, wherein the another interference filter is one of the plurality of interference filters and wherein the another interference filter is at the same time configured to pass light in a particular wavelength range for an first optical sensor and interference filter pair and reject light for a second optical sensor and interference filter pair.
 5. The system of claim 2, wherein the plurality of rejection filters is located atop one or more sets of interference filters.
 6. The system of claim 2, wherein the plurality of rejection filters has a respective top surface and a respective bottom surface and the plurality of sets of interference filters has a respective top surface and a respective bottom surface, wherein the bottom surface of the plurality of rejection filters is located proximal to the top surface of the plurality of sets of interference filters.
 7. The system of claim 2, wherein the plurality of rejection filters has a respective top surface and a respective bottom surface and the plurality of sets of interference filters has a respective top surface and a respective bottom surface, wherein the top surface of the plurality of rejection filters is located proximal to the bottom surface of the plurality of sets of interference filters.
 8. The system of claim 2, wherein a plurality of interference filters of the set of interference filters is associated with each rejection filter of the plurality of rejection filters.
 9. The system of claim 1, wherein the plurality of sets of interference filters includes a number of sets of interference filters that is less than the plurality of optical sensors.
 10. The system of claim 1, wherein each set of interference filters of the plurality of sets of interference filters is associated with a set of optical sensors of the plurality of optical sensors.
 11. The system of claim 1, wherein the set of interference filters is arranged in a pattern that further includes a plurality of organic filters.
 12. The system of claim 1, wherein the set of interference filters is arranged in a pattern that further includes a plurality of non-interference filters, wherein the non-interference filters are selected from a group that consists of at least one of organic filters and plasmonic filters.
 13. The system of claim 12, wherein the at least one organic filter is adapted to pass infrared light wavelengths.
 14. The system of claim 1, wherein at least one set of interference filters of the plurality of sets of interference filters is arranged in a pattern that further includes at least one interference filter that is respectively larger in size than at least one other interference filter in the set of interference filters.
 15. The system of claim 14, wherein the at least one interference filter that is respectively larger in size than at least one other interference filter in the set of interference filters is in an elongated rectangle shape.
 16. The system of claim 1, wherein the plurality of optical sensors includes a plurality of sets of optical sensors, wherein each set of optical sensors is arranged in a pattern that includes at least one optical sensor that is respectively larger in size than at least one other optical sensor of the set of optical sensors.
 17. The system of claim 1, wherein the pattern for the plurality of interference filters is a mosaic and the system is adapted to generate spectral information that includes a bandpass response from one or more sets of interference filters of the plurality of interference filters.
 18. The system of claim 17, wherein the bandpass response is determined based on a de-mosaicing of the mosaic.
 19. A method for imaging a scene comprises: sampling, by an imaging system, a received light spectrum from a set of spectral sensors of a plurality of sets of spectral sensors, wherein each set of spectral sensors of plurality of spectral sensors is spatially separate from every other spectral sensor of the set of spectral sensors, wherein each spectral sensor of the set of spectral sensors is associated with one or more optical sensors associated with the imaging system; comparing the received light spectrum to a plurality of light spectrum types; based on the comparing, determining whether the received light spectrum from the set of spectral sensors is representative of a light spectrum of the plurality of light spectrum types; and in response to a determination that a received light spectrum is representative of a light spectrum type, providing information for adjusting a color balance for at least a portion of the scene.
 20. The method of claim 19, further comprising: comparing the received light spectrum to a plurality of light source directionalities, wherein light source directionality is a location of the light source relative to a scene; based on the comparing, determining a light source directionality for the light source; and in response to a light source directionality determination, providing information for adjusting a color balance for the image sensor. 